Annotation of negotiation processes in joint-action dialogues

Situated dialogue corpora are invaluable resources for understanding the complex relationships among language, perception, and action. Accomplishing shared goals in the real world can often only be achieved via dynamic negotiation processes based on the interactants’ common ground. In this paper, we investigate ways of systematically capturing structural dialogue phenomena in situated goal-directed tasks through the use of annotation schemes. Specifically, we examine how dialogue structure is affected by participants updating their joint knowledge of task states by relying on various non-linguistic factors such as perceptions and actions. In contrast to entirely languagebased factors, these are typically not captured by conventional annotation schemes, although isolated relevant efforts exist. Following an overview of previous empirical evidence highlighting effects of multi-modal dialogue updates, we discuss a range of relevant dialogue corpora along with the annotation schemes that have been used to analyze them. Our brief review shows how gestures, action, intonation, perception and the like, to the extent that they are available to participants, can affect dialogue structure and directly contribute to the communicative success in situated tasks. Accordingly, current annotation schemes need to be extended to fully capture these critical additional aspects of non-linguistic dialogue phenomena, building on existing efforts as discussed in this paper.